{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 卷积神经网络\n",
    "### 从全连接层到卷积"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 不变性\n",
    "* 无论哪种方法找到这个物体，都应该和物体的位置无关。\n",
    "* 平移不变性（translation invariance）：不管检测对象出现在图像中的哪个位置，神经网络的前面几层应该对相同的图像区域具有相似的反应，即为“平移不变性”。\n",
    "* 局部性（locality）：神经网络的前面几层应该只探索输入图像中的局部区域，而不过度在意图像中相隔较远区域的关系，这就是“局部性”原则。最终，可以聚合这些局部特征，以在整个图像级别进行预测。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 多层感知机的限制\n",
    "* 为了使每个隐藏神经元都能接收到每个输入像素的信息，我们将参数从权重矩阵（如同我们先前在多层感知机中所做的那样）替换为四阶权重张量W。\n",
    "* 假设U包含偏置参数，我们可以将全连接层形式化地表示为   \n",
    "![one.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h09pmpwl5ij30cg037jrt.jpg)      \n",
    "从W到V的转换只是形式上的转换，因为在这两个四阶张量的元素之间存在一一对应的关系。索引a和b通过在正偏移和负偏移之间移动覆盖了整个图像\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "平移不变性\n",
    "* 平移不变性意味着检测对象在输入X中的平移，应该仅导致隐藏表示H中的平移。也就是说，V和U实际上不依赖于(i,j)的值，即[V]i,j,a,b=[V]a,b，并且U是一个常数。\n",
    "* 公式：（卷积）    \n",
    "![two.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h09prs4k5zj308o01edfv.jpg)  \n",
    "* 我们是在使用系数[V]a,b对位置(i,j)附近的像素(i+a,j+b)进行加权得到[H]i,j。 注意，[V]a,b的系数比[V]i,j,a,b少很多，因为前者不再依赖于图像中的位置。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "局部性\n",
    "* 为了收集用来训练参数[H]i,j的相关信息，我们不应偏离到距(i,j)很远的地方。这意味着在|a|>Δ或|b|>Δ的范围之外，我们可以设置[V]a,b=0。   \n",
    "![three.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h09pys1craj309902a74e.jpg)    \n",
    "* 上述公式是一个卷积层（convolutional layer），而卷积神经网络是包含卷积层的一类特殊的神经网络。V被称为卷积核（convolution kernel）或者滤波器（filter），它仅仅是可学习的一个层的权重。\n",
    "* 以前，多层感知机可能需要数十亿个参数来表示网络中的一层，而现在卷积神经网络通常只需要几百个参数，而且不需要改变输入或隐藏表示的维数。\n",
    "* 参数大幅减少的代价是，我们的特征现在是平移不变的，并且当确定每个隐藏活性值时，每一层只包含局部的信息。 以上所有的权重学习都将依赖于归纳偏置。\n",
    "* 当这种偏置与现实相符时，我们就能得到样本有效的模型，并且这些模型能很好地泛化到未知数据中。 但如果这偏置与现实不符时，比如当图像不满足平移不变时，我们的模型可能难以拟合我们的训练数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 卷积\n",
    "* 在数学中，两个函数（比如f,g:R^d→R）之间的“卷积”被定义为   \n",
    "![four.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h09q3ff0duj307c01dweh.jpg)     \n",
    "* 卷积是当把一个函数“翻转”并移位x时，测量f和g之间的重叠。 当为离散对象时，积分就变成求和。      \n",
    "![five.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h09q5xuuwrj30hd061wgs.jpg)     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通道\n",
    "* 实际上，图像不是二维张量，而是一个由高度、宽度和颜色组成的三维张量，比如包含1024×1024×3个像素。 前两个轴与像素的空间位置有关，而第三个轴可以看作是每个像素的多维表示。 因此，我们将X索引为[X]i,j,k。由此卷积相应地调整为[V]a,b,c，而不是[V]a,b。\n",
    "* 对于每一个空间位置，我们想要采用一组而不是一个隐藏表示。这样一组隐藏表示可以想象成一些互相堆叠的二维网格。 因此，我们可以把隐藏表示想象为一系列具有二维张量的通道（channel）。 \n",
    "* 这些通道有时也被称为特征映射（feature maps），因为每个通道都向后续层提供一组空间化的学习特征。 \n",
    "* 直观上你可以想象在靠近输入的底层，一些通道专门识别边缘，而一些通道专门识别纹理。\n",
    "![six.png](https://tva1.sinaimg.cn/large/005T39qaly1h09q9d58x0j30db02j74j.jpg)      \n",
    "其中隐藏表示H中的索引d表示输出通道，而随后的输出将继续以三维张量H作为输入进入下一个卷积层。 所以， 上述公式可以定义具有多个通道的卷积层，而其中V是该卷积层的权重。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结\n",
    "* 图像的平移不变性使我们以相同的方式处理局部图像，而不在乎它的位置。\n",
    "* 局部性意味着计算相应的隐藏表示只需一小部分局部图像像素。\n",
    "* 在图像处理中，卷积层通常比全连接层需要更少的参数，但依旧获得高效用的模型。\n",
    "* 卷积神经网络（CNN）是一类特殊的神经网络，它可以包含多个卷积层。\n",
    "* 多个输入和输出通道使模型在每个空间位置可以获取图像的多方面特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 图像卷积"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 互相关运算\n",
    "* 在卷积层中，输入张量和核张量通过互相关运算产生输出张量。\n",
    "* 输入是高度为 3 、宽度为 3 的二维张量（即形状为 3×3 ）。卷积核的高度和宽度都是 2 ，而卷积核窗口（或卷积窗口）的形状由内核的高度和宽度决定（即 2×2 ）。     \n",
    "![two.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0afbf0zn5j30kn06n767.jpg)       \n",
    "* 在二维互相关运算中，卷积窗口从输入张量的左上角开始，从左到右、从上到下滑动。 当卷积窗口滑动到新一个位置时，包含在该窗口中的部分张量与卷积核张量进行按元素相乘，得到的张量再求和得到一个单一的标量值，由此我们得出了这一位置的输出张量值。\n",
    "* 输出大小等于输入大小nh×nw减去卷积核大小kh×kw，即：    \n",
    "![three.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0afauq3ntj309f01bmx6.jpg)        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用corr2d函数中实现如上过程，该函数接受输入张量X和卷积核张量K，并返回输出张量Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "def corr2d(X,K):\n",
    "    h,w = K.shape\n",
    "    Y = torch.zeros((X.shape[0]-h+1,X.shape[1]-w+1))\n",
    "    for i in range(Y.shape[0]):\n",
    "        for j in range(Y.shape[1]):\n",
    "            Y[i,j] = (X[i:i+h,j:j+w] * K).sum()\n",
    "    return Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证二维互相关运算的输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[19., 25.],\n",
       "        [37., 43.]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.tensor([[0.0,1.0,2.0],[3.0,4.0,5.0],[6.0,7.0,8.0]])\n",
    "K = torch.tensor([[0.0,1.0],[2.0,3.0]])\n",
    "corr2d(X,K)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 卷积层\n",
    "* 卷积层对输入和卷积核权重进行互相关运算，并在添加标量偏置之后产生输出。\n",
    "* 卷积层中的两个被训练的参数是卷积核权重和标量偏置。 就像我们之前随机初始化全连接层一样，在训练基于卷积层的模型时，我们也随机初始化卷积核权重。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在__init__构造函数中，将weight和bias声明为两个模型参数。前向传播函数调用corr2d函数并添加偏置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Conv2D(nn.Module):\n",
    "    def __init__(self,kernel_size):\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.rand(kernel_size))\n",
    "        self.bias = nn.Parameter(torch.zeros(1))\n",
    "    def forward(self,x):\n",
    "        return corr2d(x,self.weight) + self.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "高度和宽度分别为h和w的卷积核可以被称为h×w卷积或h×w卷积核"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  图像中目标的边缘检测\n",
    "简单应用：通过找到像素变化的位置，来检测图像中不同颜色的边缘。\n",
    "*  首先，我们构造一个 6×8 像素的黑白图像。中间四列为黑色（ 0 ），其余像素为白色（ 1 ）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 0., 0., 0., 0., 1., 1.],\n",
       "        [1., 1., 0., 0., 0., 0., 1., 1.],\n",
       "        [1., 1., 0., 0., 0., 0., 1., 1.],\n",
       "        [1., 1., 0., 0., 0., 0., 1., 1.],\n",
       "        [1., 1., 0., 0., 0., 0., 1., 1.],\n",
       "        [1., 1., 0., 0., 0., 0., 1., 1.]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.ones((6,8))\n",
    "X[:,2:6] = 0\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 接下来，我们构造一个高度为 1 、宽度为 2 的卷积核K。当进行互相关运算时，如果水平相邻的两元素相同，则输出为零，否则输出为非零。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = torch.tensor([[1.0,-1.0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 我们对参数X（输入）和K（卷积核）执行互相关运算。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  0.,  0.,  0., -1.,  0.],\n",
       "        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],\n",
       "        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],\n",
       "        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],\n",
       "        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],\n",
       "        [ 0.,  1.,  0.,  0.,  0., -1.,  0.]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y = corr2d(X,K)\n",
    "Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输出Y中的1代表从白色到黑色的边缘，-1代表从黑色到白色的边缘，其他情况的输出为 0 。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 我们将输入的二维图像转置，再进行如上的互相关运算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corr2d(X.t(),K)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之前检测到的垂直边缘消失了。这个卷积核K只可以检测垂直边缘，无法检测水平边缘。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 学习卷积核"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学习有X生成Y的卷积核\n",
    "* 我们先构造一个卷积层，并将其卷积核初始化为随机张量。\n",
    "* 接下来，在每次迭代中，我们比较Y与卷积层输出的平方误差，然后计算梯度来更新卷积核。\n",
    "* 为了简单起见，我们在此使用内置的二维卷积层，并忽略偏置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 2,loss 9.776\n",
      "epoch 4,loss 1.688\n",
      "epoch 6,loss 0.303\n",
      "epoch 8,loss 0.059\n",
      "epoch 10,loss 0.013\n"
     ]
    }
   ],
   "source": [
    "# 构造一个二维卷积层，它具有一个通道和形状为（1，2）的卷积核\n",
    "conv2d = nn.Conv2d(1,1,kernel_size=(1,2),bias=False)\n",
    "\n",
    "# 这个二维卷积层使用四维输入和输出格式（批量大小、通道、高度、宽度）\n",
    "# 其中批量大小和通道数都为1\n",
    "X = X.reshape((1,1,6,8))\n",
    "Y = Y.reshape((1,1,6,7))\n",
    "lr = 3e-2\n",
    "\n",
    "for i in range(10):\n",
    "    Y_hat = conv2d(X)\n",
    "    l = (Y_hat - Y) ** 2\n",
    "    conv2d.zero_grad()\n",
    "    l.sum().backward()\n",
    "    # 迭代卷积核\n",
    "    conv2d.weight.data[:] -= lr*conv2d.weight.grad\n",
    "    if (i+1) % 2 == 0:\n",
    "        print(f'epoch {i+1},loss {l.sum():.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "所学的卷积核的权重张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9916, -0.9762]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv2d.weight.data.reshape((1,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 互相关和卷积\n",
    "* 由于卷积核是从数据中学习到的，因此无论这些层执行严格的卷积运算还是互相关运算，卷积层的输出都不会受到影响。\n",
    "*  假设其他条件不变，当这个层执行严格的卷积时，学习的卷积核 K′ 在水平和垂直翻转之后将与 K 相同。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 特征映射和感受野\n",
    "* 输出的卷积层有时被称为特征映射（feature map），因为它可以被视为一个输入映射到下一层的空间维度的转换器。\n",
    "* 在卷积神经网络中，对于某一层的任意元素 x ，其感受野（receptive field）是指在前向传播期间可能影响 x 计算的所有元素（来自所有先前层）。\n",
    "* 感受野可能大于输入的实际大小。\n",
    "* 当一个特征图中的任意元素需要检测更广区域的输入特征时，我们可以构建一个更深的网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 小结\n",
    "* 二维卷积层的核心计算是二维互相关运算。最简单的形式是，对二维输入数据和卷积核执行互相关操作，然后添加一个偏置。\n",
    "* 我们可以设计一个卷积核来检测图像的边缘。\n",
    "* 我们可以从数据中学习卷积核的参数。\n",
    "* 学习卷积核时，无论用严格卷积运算或互相关运算，卷积层的输出不会受太大影响。\n",
    "* 当需要检测输入特征中更广区域时，我们可以构建一个更深的卷积网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 填充和步幅"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 卷积的输出形状取决于输入形状和卷积核的形状。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 填充\n",
    "* 在应用多层卷积时，我们常常丢失边缘像素。\n",
    "* 由于我们通常使用小卷积核，因此对于任何单个卷积，我们可能只会丢失几个像素。 但随着我们应用许多连续卷积层，累积丢失的像素数就多了。\n",
    "* 填充（padding）：在输入图像的边界填充元素（通常填充元素是）。     \n",
    "![1.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0bymhkvjzj30eu07igmr.jpg)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在许多情况下，我们需要设置ph = kh -1和pw = kw -1，使输入和输出具有相同的高度和宽度。    \n",
    "卷积神经网络中卷积核的高度和宽度通常为奇数，例如1、3、5或7。 选择奇数的好处是，保持空间维度的同时，我们可以在顶部和底部填充相同数量的行，在左侧和右侧填充相同数量的列。     \n",
    "对于任何二维张量X，当满足： 1. 卷积核的大小是奇数； 2. 所有边的填充行数和列数相同； 3. 输出与输入具有相同高度和宽度 则可以得出：输出Y[i, j]是通过以输入X[i, j]为中心，与卷积核进行互相关计算得到的。    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建一个高度和宽度为3的二维卷积层，并在所有侧边填充1个像素。给定高度和宽度为8的输入，则输出的高度和宽度也是8。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([8, 8])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch \n",
    "from torch import nn\n",
    "\n",
    "# 我们定义了一个计算卷积层的函数\n",
    "# 此函数初始化卷积层权重，并对输入和输出提高和缩减相应的维数\n",
    "def comp_conv2d(conv2d,X):\n",
    "    # 这里的（1，1）表示大小和通道数都是1\n",
    "    X = X.reshape((1,1) + X.shape)\n",
    "    Y = conv2d(X)\n",
    "    # 省略前两个维度，批量大小和通道\n",
    "    return Y.reshape(Y.shape[2:])\n",
    "\n",
    "# 请注意，这里每边填充1行或1列，因此总共添加了2行或2列\n",
    "conv2d = nn.Conv2d(1,1,kernel_size=3,padding=1)\n",
    "X = torch.rand(size=(8,8))\n",
    "comp_conv2d(conv2d,X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当卷积核的高度和宽度不同时，我们可以填充不同的高度和宽度，使输出和输入具有相同的高度和宽度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([8, 8])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv2d = nn.Conv2d(1,1,kernel_size=(5,3),padding=(2,1))\n",
    "comp_conv2d(conv2d,X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 步幅\n",
    "* 有时候为了高效计算或是缩减采样次数，卷积窗口可以跳过中间位置，每次滑动多个元素。\n",
    "* 将每次滑动元素的数量称为步幅（stride）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了计算输出中第一列的第二个元素和第一行的第二个元素，卷积窗口分别向下滑动三行和向右滑动两列。但是，当卷积窗口继续向右滑动两列时，没有输出，因为输入元素无法填充窗口（除非我们添加另一列填充）。    \n",
    "![2.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0bza4op6fj30eq07ejsu.jpg)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当垂直步幅为Sh、水平步幅为Sw时，输出形状为      \n",
    "![3.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0bzbab6gkj30cf016q36.jpg)    \n",
    "如果输入的高度和宽度可以被垂直和水平步幅整除，则输出形状将为(nh/sh) X (nw/sw)。     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将高度和宽度的步幅设置为2，从而将输入的高度和宽度减半."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 4])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv2d = nn.Conv2d(1,1,kernel_size=3,padding=1,stride=2)\n",
    "comp_conv2d(conv2d,X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "稍微复杂的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 2])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conv2d = nn.Conv2d(1,1,kernel_size=(3,5),padding=(0,1),stride=(3,4))\n",
    "comp_conv2d(conv2d,X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 小结\n",
    "* 填充可以增加输出的高度和宽度。这常用来使输出与输入具有相同的高和宽。\n",
    "* 步幅可以减小输出的高和宽，例如输出的高和宽仅为输入的高和宽的1/n（n是一个大于1的整数）。\n",
    "* 填充和步幅可用于有效地调整数据的维度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多输入多输出通道\n",
    "当我们添加通道时，我们的输入和隐藏的表示都变成了三维张量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 多输入通道\n",
    "* 当输入包含多个通道时，需要构造一个与输入数据具有相同输入通道数的卷积核，以便与输入数据进行互相关运算。\n",
    "* 然而，当Ci > 1时，我们卷积核的每个输入通道将包含形状为Kh * Kw的张量。将这些张量Ci连结在一起可以得到形状为Ci * Kh * Kw的卷积核.\n",
    "* 由于输入和卷积核都有Ci个通道，我们可以对每个通道输入的二维张量和卷积核的二维张量进行互相关运算，再对通道求和（将Ci的结果相加）得到二维张量。      \n",
    "![4.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0c071w11fj30ha084wg2.jpg)    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "多输入通道互相关操作：简而言之，我们所做的就是对每个通道执行互相关操作，然后将结果相加。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from d2l import torch as d2l\n",
    "\n",
    "def corr2d_multi_in(X,K):\n",
    "    # 先遍历“X”和“K”的第0维度，再把它们加在一起\n",
    "    return sum(d2l.corr2d(x,k) for x,k in zip(X,K))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证互相关运算的输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 56.,  72.],\n",
       "        [104., 120.]])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],\n",
    "               [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])\n",
    "K = torch.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])\n",
    "\n",
    "corr2d_multi_in(X, K)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 多输出通道\n",
    "* 在最流行的神经网络架构中，随着神经网络层数的加深，我们常会增加输出通道的维数，通过减少空间分辨率以获得更大的通道深度。\n",
    "* 因为每个通道不是独立学习的，而是为了共同使用而优化的。因此，多输出通道并不仅是学习多个单通道的检测器。\n",
    "* 为了获得多个通道的输出，我们可以为每个输出通道创建一个形状为Ci * Kh * Kw的卷积核张量，这样卷积核的形状是Co * Ci * Kh * Kw。\n",
    "* 在互相关运算中，每个输出通道先获取所有输入通道，再以对应该输出通道的卷积核计算出结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实现一个计算多个通道的输出的互相关函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def corr2d_multi_in_out(X,K):\n",
    "    # 迭代“K”的第0个维度，每次都对输入“X”执行互相关运算\n",
    "    # 最后将所有的结果都叠加在一起\n",
    "    return torch.stack([corr2d_multi_in(X,k) for k in K],0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过将核张量K与K+1（K中每个元素加1）和K+2连接起来，构造了一个具有3个输出通道的卷积核。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2, 2, 2])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "K = torch.stack((K,K+1,K+2),0)\n",
    "K.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对输入张量X与卷积核张量K执行互相关运算;现在的输出包含3个通道，第一个通道的结果与先前输入张量X和多输入单输出通道的结果一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 56.,  72.],\n",
       "         [104., 120.]],\n",
       "\n",
       "        [[ 76., 100.],\n",
       "         [148., 172.]],\n",
       "\n",
       "        [[ 96., 128.],\n",
       "         [192., 224.]]])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corr2d_multi_in_out(X,K)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 X 1卷积层\n",
    "* 卷积的本质是有效提取相邻像素间的相关特征，而1X1卷积显然没有此作用。 尽管如此，仍然十分流行，经常包含在复杂深层网络的设计中。\n",
    "* 因为使用了最小窗口，1X1卷积失去了卷积层的特有能力——在高度和宽度维度上，识别相邻元素间相互作用的能力。 \n",
    "* 输入和输出具有相同的高度和宽度，输出中的每个元素都是从输入图像中同一位置的元素的线性组合。\n",
    "* 我们可以将1X1卷积层看作是在每个像素位置应用的全连接层，以Ci个输入值转换Co为个输出值。 \n",
    "* 1X1卷积层需要的权重维度为Co X Ci，再额外加上一个偏置。    \n",
    "![5.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0c0rccrwdj30gj06ztaf.jpg)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用全连接层实现1X1卷积"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def corr2d_multi_in_out_1X1(X,K):\n",
    "    c_i,h,w = X.shape\n",
    "    c_o = K.shape[0]\n",
    "    X = X.reshape((c_i,h*w))\n",
    "    K = K.reshape((c_o,c_i))\n",
    "    # 全连接层中的矩阵乘法\n",
    "    Y = torch.matmul(K,X)\n",
    "    return Y.reshape((c_o,h,w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用一些样本数据来验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.normal(0,1,(3,3,3))\n",
    "K = torch.normal(0,1,(2,3,1,1))\n",
    "\n",
    "Y1 = corr2d_multi_in_out_1X1(X,K)\n",
    "Y2 = corr2d_multi_in_out(X,K)\n",
    "assert float(torch.abs(Y1 - Y2).sum()) < 1e-6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 小结\n",
    "* 多输入多输出通道可以用来扩展卷积层的模型。\n",
    "* 当以每像素为基础应用时，1X1卷积层相当于全连接层。\n",
    "* 1X1卷积层通常用于调整网络层的通道数量和控制模型复杂性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 汇聚层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 通过逐渐聚合信息，生成越来越粗糙的映射，最终实现学习全局表示的目标，同时将卷积图层的所有优势保留在中间层。\n",
    "* 当检测较底层的特征时，我们通常希望这些特征保持某种程度上的平移不变性。\n",
    "* 而在现实中，随着拍摄角度的移动，任何物体几乎不可能发生在同一像素上。即使用三脚架拍摄一个静止的物体，由于快门的移动而引起的相机振动，可能会使所有物体左右移动一个像素。\n",
    "* 汇聚层的目的：降低卷积层对位置的敏感性，同时降低对空间降采样表示的敏感性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 最大汇聚层和平均汇聚层\n",
    "* 与卷积层类似，汇聚层运算符由一个固定形状的窗口组成，该窗口根据其步幅大小在输入的所有区域上滑动，为固定形状窗口（有时称为汇聚窗口）遍历的每个位置计算一个输出。\n",
    "* 池运算是确定性的，我们通常计算汇聚窗口中所有元素的最大值或平均值。这些操作分别称为最大汇聚层（maximum pooling）和平均汇聚层（average pooling）。\n",
    "* 汇聚窗口从输入张量的左上角开始，从左往右、从上往下的在输入张量内滑动。在汇聚窗口到达的每个位置，它计算该窗口中输入子张量的最大值或平均值。计算最大值或平均值是取决于使用了最大汇聚层还是平均汇聚层。      \n",
    "![1.png](https://image.baidu.com/search/down?url=https://tva1.sinaimg.cn/large/005T39qaly1h0cvwufihxj30hk05xdhf.jpg)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用2X2最大汇聚层，即使在高度或宽度上移动一个元素，卷积层仍然可以识别到模式。\n",
    "* 下面的代码中的pool2d函数，我们实现汇聚层的前向传播"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "def pool2d(X,pool_size,mode='max'):\n",
    "    p_h,p_w = pool_size\n",
    "    Y = torch.zeros(X.shape[0] - p_h + 1,X.shape[1] - p_w + 1)\n",
    "    for i in range(Y.shape[0]):\n",
    "        for j in range(Y.shape[1]):\n",
    "            if mode == 'max':\n",
    "                Y[i,j] = X[i:i+p_h,j:j+p_w].max()\n",
    "            elif mode == 'avg':\n",
    "                Y[i,j] = X[i:i+p_h,j:j+p_w].mean()\n",
    "    return Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证二维最大汇聚层的输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[4., 5.],\n",
       "        [7., 8.]])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.tensor([[0.0,1.0,2.0],[3.0,4.0,5.0],[6.0,7.0,8.0]])\n",
    "pool2d(X,(2,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证平均汇聚层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 3.],\n",
       "        [5., 6.]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool2d(X,(2,2),'avg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 填充和步幅\n",
    "演示汇聚层中填充和步幅的使用\n",
    "* 首先构造了一个输入张量X，它有四个维度，其中样本数和通道数都是1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[ 0.,  1.,  2.,  3.],\n",
       "          [ 4.,  5.,  6.,  7.],\n",
       "          [ 8.,  9., 10., 11.],\n",
       "          [12., 13., 14., 15.]]]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.arange(16,dtype=torch.float32).reshape(1,1,4,4)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 深度学习框架中的步幅与汇聚窗口的大小相同。 因此，如果我们使用形状为(3, 3)的汇聚窗口，那么默认情况下，我们得到的步幅形状为(3, 3)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[10.]]]])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool2d = nn.MaxPool2d(3)\n",
    "pool2d(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "填充和步幅可以手动设定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[ 5.,  7.],\n",
       "          [13., 15.]]]])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool2d = nn.MaxPool2d(3,padding=1,stride=2)\n",
    "pool2d(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以设定一个任意大小的矩形汇聚窗口，并分别设定填充和步幅的高度和宽度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[ 5.,  7.],\n",
       "          [13., 15.]]]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool2d = nn.MaxPool2d((2,3),stride=(2,3),padding=(0,1))\n",
    "pool2d(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 多个通道\n",
    "在处理多通道输入数据时，汇聚层在每个输入通道上单独运算，而不是像卷积层一样在通道上对输入进行汇总。 这意味着汇聚层的输出通道数与输入通道数相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将在通道维度上连结张量X和X + 1，以构建具有2个通道的输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[ 0.,  1.,  2.,  3.],\n",
       "          [ 4.,  5.,  6.,  7.],\n",
       "          [ 8.,  9., 10., 11.],\n",
       "          [12., 13., 14., 15.]],\n",
       "\n",
       "         [[ 1.,  2.,  3.,  4.],\n",
       "          [ 5.,  6.,  7.,  8.],\n",
       "          [ 9., 10., 11., 12.],\n",
       "          [13., 14., 15., 16.]]]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.cat((X,X+1),1)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[ 5.,  7.],\n",
       "          [13., 15.]],\n",
       "\n",
       "         [[ 6.,  8.],\n",
       "          [14., 16.]]]])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool2d = nn.MaxPool2d(3,padding=1,stride=2)\n",
    "pool2d(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 小结\n",
    "* 对于给定输入元素，最大汇聚层会输出该窗口内的最大值，平均汇聚层会输出该窗口内的平均值。\n",
    "* 汇聚层的主要优点之一是减轻卷积层对位置的过度敏感。\n",
    "* 我们可以指定汇聚层的填充和步幅。\n",
    "* 使用最大汇聚层以及大于1的步幅，可减少空间维度（如高度和宽度）。\n",
    "* 汇聚层的输出通道数与输入通道数相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 卷积神经网络（LeNet）\n",
    "* 用卷积层代替全连接层的另一个好处是：模型更简洁、所需的参数更少。\n",
    "* LeNet（LeNet-5）由两个部分组成：\n",
    "    * 卷积编码器：由两个卷积层组成;\n",
    "    * 全连接层密集块：由三个全连接层组成。\n",
    "![LeNet.png](https://tva1.sinaimg.cn/large/005T39qaly1h0cwsc3l0uj30i5079wg2.jpg)    \n",
    "* 每个卷积块中的基本单元是一个卷积层、一个sigmoid激活函数和平均汇聚层。\n",
    "* 卷积的输出形状由批量大小、通道数、高度、宽度决定。\n",
    "* 为了将卷积块的输出传递给稠密块，我们必须在小批量中展平每个样本。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实现LeNet:实例化一个Sequential块并将需要的层连接在一起"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "net = nn.Sequential(\n",
    "    nn.Conv2d(1,6,kernel_size=5,padding=2),\n",
    "    nn.Sigmoid(),\n",
    "    nn.AvgPool2d(kernel_size=2,stride=2),\n",
    "    nn.Conv2d(6,16,kernel_size=5),\n",
    "    nn.Sigmoid(),\n",
    "    nn.AvgPool2d(kernel_size=2,stride=2),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(16*5*5,120),\n",
    "    nn.Sigmoid(),\n",
    "    nn.Linear(120,84),\n",
    "    nn.Sigmoid(),\n",
    "    nn.Linear(84,10)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们对原始模型做了一点小改动，去掉了最后一层的高斯激活。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将一个大小为的单通道（黑白）图像通过LeNet。通过在每一层打印输出的形状，我们可以检查模型.    \n",
    "![2.png](https://tva1.sinaimg.cn/large/005T39qaly1h0cx38nh1lj307y0czdha.jpg)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conv2d output shape:\t torch.Size([1, 6, 28, 28])\n",
      "Sigmoid output shape:\t torch.Size([1, 6, 28, 28])\n",
      "AvgPool2d output shape:\t torch.Size([1, 6, 14, 14])\n",
      "Conv2d output shape:\t torch.Size([1, 16, 10, 10])\n",
      "Sigmoid output shape:\t torch.Size([1, 16, 10, 10])\n",
      "AvgPool2d output shape:\t torch.Size([1, 16, 5, 5])\n",
      "Flatten output shape:\t torch.Size([1, 400])\n",
      "Linear output shape:\t torch.Size([1, 120])\n",
      "Sigmoid output shape:\t torch.Size([1, 120])\n",
      "Linear output shape:\t torch.Size([1, 84])\n",
      "Sigmoid output shape:\t torch.Size([1, 84])\n",
      "Linear output shape:\t torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "X = torch.rand(size=(1,1,28,28),dtype=torch.float32)\n",
    "for layer in net:\n",
    "    X = layer(X)\n",
    "    print(layer.__class__.__name__,'output shape:\\t',X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " 第一个卷积层使用2个像素的填充，来补偿5X5卷积核导致的特征减少。 相反，第二个卷积层没有填充，因此高度和宽度都减少了4个像素。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 模型训练\n",
    "LeNet在Fashion-MNIST数据集上的表现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 256\n",
    "train_iter,test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然卷积神经网络的参数较少，但与深度的多层感知机相比，它们的计算成本仍然很高，因为每个参数都参与更多的乘法。    \n",
    "由于完整的数据集位于内存中，因此在模型使用GPU计算数据集之前，我们需要将其复制到显存中。     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_accuracy_gpu(net,data_iter,device=None):\n",
    "    # 使用GPU计算模型在数据集上的精度\n",
    "    if isinstance(net,nn.Module):\n",
    "        net.eval() # 设置评估模式\n",
    "        if not device:\n",
    "            device = next(iter(net.parameters())).device\n",
    "    # 正确预测的数量，总预测的数量\n",
    "    metric = d2l.Accumulator(2)\n",
    "    with torch.no_grad():\n",
    "        for X,y in data_iter:\n",
    "            if isinstance(X,list):\n",
    "                # BERT微调所需\n",
    "                X = [x.to(device) for x in X]\n",
    "            else:\n",
    "                X = X.to(device)\n",
    "            y = y.to(device)\n",
    "            metric.add(d2l.accuracy(net(X),y),y.numel())\n",
    "    return metric[0]/metric[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Xavier随机初始化模型参数。 与全连接层一样，我们使用交叉熵损失函数和小批量随机梯度下降。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@save\n",
    "def train_ch6(net, train_iter, test_iter, num_epochs, lr, device):\n",
    "    \"\"\"用GPU训练模型(在第六章定义)\"\"\"\n",
    "    def init_weights(m):\n",
    "        if type(m) == nn.Linear or type(m) == nn.Conv2d:\n",
    "            nn.init.xavier_uniform_(m.weight)\n",
    "    net.apply(init_weights)\n",
    "    print('training on', device)\n",
    "    net.to(device)\n",
    "    optimizer = torch.optim.SGD(net.parameters(), lr=lr)\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "    animator = d2l.Animator(xlabel='epoch', xlim=[1, num_epochs],\n",
    "                            legend=['train loss', 'train acc', 'test acc'])\n",
    "    timer, num_batches = d2l.Timer(), len(train_iter)\n",
    "    for epoch in range(num_epochs):\n",
    "        # 训练损失之和，训练准确率之和，样本数\n",
    "        metric = d2l.Accumulator(3)\n",
    "        net.train()\n",
    "        for i, (X, y) in enumerate(train_iter):\n",
    "            timer.start()\n",
    "            optimizer.zero_grad()\n",
    "            X, y = X.to(device), y.to(device)\n",
    "            y_hat = net(X)\n",
    "            l = loss(y_hat, y)\n",
    "            l.backward()\n",
    "            optimizer.step()\n",
    "            with torch.no_grad():\n",
    "                metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0])\n",
    "            timer.stop()\n",
    "            train_l = metric[0] / metric[2]\n",
    "            train_acc = metric[1] / metric[2]\n",
    "            if (i + 1) % (num_batches // 5) == 0 or i == num_batches - 1:\n",
    "                animator.add(epoch + (i + 1) / num_batches,\n",
    "                             (train_l, train_acc, None))\n",
    "        test_acc = evaluate_accuracy_gpu(net, test_iter)\n",
    "        animator.add(epoch + 1, (None, None, test_acc))\n",
    "    print(f'loss {train_l:.3f}, train acc {train_acc:.3f}, '\n",
    "          f'test acc {test_acc:.3f}')\n",
    "    print(f'{metric[2] * num_epochs / timer.sum():.1f} examples/sec '\n",
    "          f'on {str(device)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练和评估LeNet-5模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.480, train acc 0.819, test acc 0.815\n",
      "12150.4 examples/sec on cuda:0\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 238.965625 180.65625\" width=\"238.965625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2022-03-17T15:58:31.509858</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.3.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 238.965625 180.65625 \r\n",
       "L 238.965625 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 30.103125 143.1 \r\n",
       "L 225.403125 143.1 \r\n",
       "L 225.403125 7.2 \r\n",
       "L 30.103125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 51.803125 143.1 \r\n",
       "L 51.803125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m8c60c7f663\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.803125\" xlink:href=\"#m8c60c7f663\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(48.621875 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-50\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 95.203125 143.1 \r\n",
       "L 95.203125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"95.203125\" xlink:href=\"#m8c60c7f663\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(92.021875 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-52\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 138.603125 143.1 \r\n",
       "L 138.603125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"138.603125\" xlink:href=\"#m8c60c7f663\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(135.421875 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-54\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 182.003125 143.1 \r\n",
       "L 182.003125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"182.003125\" xlink:href=\"#m8c60c7f663\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(178.821875 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-56\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 225.403125 143.1 \r\n",
       "L 225.403125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"225.403125\" xlink:href=\"#m8c60c7f663\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 10 -->\r\n",
       "      <g transform=\"translate(219.040625 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-49\"/>\r\n",
       "        <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-48\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epoch -->\r\n",
       "     <g transform=\"translate(112.525 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-101\"/>\r\n",
       "       <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-112\"/>\r\n",
       "       <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-111\"/>\r\n",
       "       <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-99\"/>\r\n",
       "       <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-104\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 30.103125 142.335066 \r\n",
       "L 225.403125 142.335066 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m6e57cf46e8\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m6e57cf46e8\" y=\"142.335066\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <g transform=\"translate(7.2 146.134285)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 10.6875 12.40625 \r\n",
       "L 21 12.40625 \r\n",
       "L 21 0 \r\n",
       "L 10.6875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-46\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 30.103125 115.201206 \r\n",
       "L 225.403125 115.201206 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m6e57cf46e8\" y=\"115.201206\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <g transform=\"translate(7.2 119.000425)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-53\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 30.103125 88.067346 \r\n",
       "L 225.403125 88.067346 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m6e57cf46e8\" y=\"88.067346\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <g transform=\"translate(7.2 91.866565)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 30.103125 60.933487 \r\n",
       "L 225.403125 60.933487 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m6e57cf46e8\" y=\"60.933487\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 1.5 -->\r\n",
       "      <g transform=\"translate(7.2 64.732705)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#p190208590a)\" d=\"M 30.103125 33.799627 \r\n",
       "L 225.403125 33.799627 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m6e57cf46e8\" y=\"33.799627\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 2.0 -->\r\n",
       "      <g transform=\"translate(7.2 37.598845)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_21\">\r\n",
       "    <path clip-path=\"url(#p190208590a)\" d=\"M 12.743125 13.377273 \r\n",
       "L 17.083125 15.051613 \r\n",
       "L 21.423125 15.703643 \r\n",
       "L 25.763125 16.072687 \r\n",
       "L 30.103125 16.300316 \r\n",
       "L 34.443125 17.29087 \r\n",
       "L 38.783125 17.304933 \r\n",
       "L 43.123125 17.421502 \r\n",
       "L 47.463125 19.821057 \r\n",
       "L 51.803125 28.320444 \r\n",
       "L 56.143125 78.265053 \r\n",
       "L 60.483125 81.517973 \r\n",
       "L 64.823125 84.299378 \r\n",
       "L 69.163125 86.273064 \r\n",
       "L 73.503125 88.082074 \r\n",
       "L 77.843125 98.895471 \r\n",
       "L 82.183125 98.99467 \r\n",
       "L 86.523125 99.986169 \r\n",
       "L 90.863125 100.807881 \r\n",
       "L 95.203125 101.443218 \r\n",
       "L 99.543125 105.400772 \r\n",
       "L 103.883125 105.575715 \r\n",
       "L 108.223125 106.196779 \r\n",
       "L 112.563125 106.747943 \r\n",
       "L 116.903125 107.109348 \r\n",
       "L 121.243125 107.010678 \r\n",
       "L 125.583125 107.860605 \r\n",
       "L 129.923125 108.426208 \r\n",
       "L 134.263125 108.844991 \r\n",
       "L 138.603125 109.152015 \r\n",
       "L 142.943125 110.669673 \r\n",
       "L 147.283125 111.067314 \r\n",
       "L 151.623125 111.42562 \r\n",
       "L 155.963125 111.624525 \r\n",
       "L 160.303125 111.753892 \r\n",
       "L 164.643125 113.485543 \r\n",
       "L 168.983125 113.148342 \r\n",
       "L 173.323125 113.408005 \r\n",
       "L 177.663125 113.548191 \r\n",
       "L 182.003125 113.569212 \r\n",
       "L 186.343125 114.690443 \r\n",
       "L 190.683125 114.739847 \r\n",
       "L 195.023125 114.774907 \r\n",
       "L 199.363125 114.874712 \r\n",
       "L 203.703125 115.057487 \r\n",
       "L 208.043125 114.959079 \r\n",
       "L 212.383125 115.79946 \r\n",
       "L 216.723125 115.921941 \r\n",
       "L 221.063125 116.136546 \r\n",
       "L 225.403125 116.313082 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_22\">\r\n",
       "    <path clip-path=\"url(#p190208590a)\" d=\"M 12.743125 136.922727 \r\n",
       "L 17.083125 136.857328 \r\n",
       "L 21.423125 136.806963 \r\n",
       "L 25.763125 136.853945 \r\n",
       "L 30.103125 136.886587 \r\n",
       "L 34.443125 136.791929 \r\n",
       "L 38.783125 136.776143 \r\n",
       "L 43.123125 136.461175 \r\n",
       "L 47.463125 135.056598 \r\n",
       "L 51.803125 132.205996 \r\n",
       "L 56.143125 114.438969 \r\n",
       "L 60.483125 112.94832 \r\n",
       "L 64.823125 111.650111 \r\n",
       "L 69.163125 110.848784 \r\n",
       "L 73.503125 110.025875 \r\n",
       "L 77.843125 105.030519 \r\n",
       "L 82.183125 105.138766 \r\n",
       "L 86.523125 104.720813 \r\n",
       "L 90.863125 104.37089 \r\n",
       "L 95.203125 104.143254 \r\n",
       "L 99.543125 102.626539 \r\n",
       "L 103.883125 102.516037 \r\n",
       "L 108.223125 102.208586 \r\n",
       "L 112.563125 101.954507 \r\n",
       "L 116.903125 101.809742 \r\n",
       "L 121.243125 101.819198 \r\n",
       "L 125.583125 101.544071 \r\n",
       "L 129.923125 101.259923 \r\n",
       "L 134.263125 101.099808 \r\n",
       "L 138.603125 100.931509 \r\n",
       "L 142.943125 100.272171 \r\n",
       "L 147.283125 100.134607 \r\n",
       "L 151.623125 99.93841 \r\n",
       "L 155.963125 99.823398 \r\n",
       "L 160.303125 99.797314 \r\n",
       "L 164.643125 99.149111 \r\n",
       "L 168.983125 99.309226 \r\n",
       "L 173.323125 99.174669 \r\n",
       "L 177.663125 99.137835 \r\n",
       "L 182.003125 99.093643 \r\n",
       "L 186.343125 98.522181 \r\n",
       "L 190.683125 98.515416 \r\n",
       "L 195.023125 98.522181 \r\n",
       "L 199.363125 98.484972 \r\n",
       "L 203.703125 98.430672 \r\n",
       "L 208.043125 98.621408 \r\n",
       "L 212.383125 98.201951 \r\n",
       "L 216.723125 98.113249 \r\n",
       "L 221.063125 98.001244 \r\n",
       "L 225.403125 97.898848 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_23\">\r\n",
       "    <path clip-path=\"url(#p190208590a)\" d=\"M 30.103125 136.908294 \r\n",
       "L 51.803125 117.361062 \r\n",
       "L 73.503125 105.167105 \r\n",
       "L 95.203125 109.079808 \r\n",
       "L 116.903125 101.525741 \r\n",
       "L 138.603125 102.616522 \r\n",
       "L 160.303125 101.265256 \r\n",
       "L 182.003125 100.087646 \r\n",
       "L 203.703125 98.606138 \r\n",
       "L 225.403125 98.123155 \r\n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 30.103125 143.1 \r\n",
       "L 30.103125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 225.403125 143.1 \r\n",
       "L 225.403125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 30.103125 143.1 \r\n",
       "L 225.403125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 30.103125 7.2 \r\n",
       "L 225.403125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 140.634375 59.234375 \r\n",
       "L 218.403125 59.234375 \r\n",
       "Q 220.403125 59.234375 220.403125 57.234375 \r\n",
       "L 220.403125 14.2 \r\n",
       "Q 220.403125 12.2 218.403125 12.2 \r\n",
       "L 140.634375 12.2 \r\n",
       "Q 138.634375 12.2 138.634375 14.2 \r\n",
       "L 138.634375 57.234375 \r\n",
       "Q 138.634375 59.234375 140.634375 59.234375 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_24\">\r\n",
       "     <path d=\"M 142.634375 20.298438 \r\n",
       "L 162.634375 20.298438 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_25\"/>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- train loss -->\r\n",
       "     <g transform=\"translate(170.634375 23.798438)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-116\"/>\r\n",
       "       <path d=\"M 41.109375 46.296875 \r\n",
       "Q 39.59375 47.171875 37.8125 47.578125 \r\n",
       "Q 36.03125 48 33.890625 48 \r\n",
       "Q 26.265625 48 22.1875 43.046875 \r\n",
       "Q 18.109375 38.09375 18.109375 28.8125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 20.953125 51.171875 25.484375 53.578125 \r\n",
       "Q 30.03125 56 36.53125 56 \r\n",
       "Q 37.453125 56 38.578125 55.875 \r\n",
       "Q 39.703125 55.765625 41.0625 55.515625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-114\"/>\r\n",
       "       <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-97\"/>\r\n",
       "       <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-105\"/>\r\n",
       "       <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-110\"/>\r\n",
       "       <path id=\"DejaVuSans-32\"/>\r\n",
       "       <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-108\"/>\r\n",
       "       <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-115\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-108\"/>\r\n",
       "      <use x=\"292.333984\" xlink:href=\"#DejaVuSans-111\"/>\r\n",
       "      <use x=\"353.515625\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"405.615234\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_26\">\r\n",
       "     <path d=\"M 142.634375 34.976562 \r\n",
       "L 162.634375 34.976562 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_27\"/>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- train acc -->\r\n",
       "     <g transform=\"translate(170.634375 38.476562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\r\n",
       "      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"325.830078\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"380.810547\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_28\">\r\n",
       "     <path d=\"M 142.634375 49.654688 \r\n",
       "L 162.634375 49.654688 \r\n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_29\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- test acc -->\r\n",
       "     <g transform=\"translate(170.634375 53.154688)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\r\n",
       "      <use x=\"192.041016\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      <use x=\"223.828125\" xlink:href=\"#DejaVuSans-97\"/>\r\n",
       "      <use x=\"285.107422\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "      <use x=\"340.087891\" xlink:href=\"#DejaVuSans-99\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p190208590a\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"30.103125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr,num_epochs = 0.9,10\n",
    "train_ch6(net,train_iter,test_iter,num_epochs,lr,d2l.try_gpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 小结\n",
    "* 卷积神经网络（CNN）是一类使用卷积层的网络。\n",
    "* 在卷积神经网络中，我们组合使用卷积层、非线性激活函数和汇聚层。\n",
    "* 为了构造高性能的卷积神经网络，我们通常对卷积层进行排列，逐渐降低其表示的空间分辨率，同时增加通道数。\n",
    "* 在传统的卷积神经网络中，卷积块编码得到的表征在输出之前需由一个或多个全连接层进行处理。\n",
    "* LeNet是最早发布的卷积神经网络之一。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "6ca3aa8ecd0b7bddf142c4852696022db00ef254c6fe98093bb8bf9134bdc29e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
